SOLR-18060: Add Prometheus metrics to CrossDC Consumer. by sigram · Pull Request #4063 · apache/solr

sigram · 2026-01-19T14:10:40Z

This PR replaces Dropwizard JSON metrics with Prometheus metrics in the CrossDC Consumer, using directly the Prometheus client_java API. It also removes the Dropwizard dependency.

mlbiscoc

Can you post a sample of all these metrics? Either dump it here or in a txt file? It would be easier to review the names and labels on the metrics.

gradle/libs.versions.toml

solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaIntegrationTest.java

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

mlbiscoc · 2026-01-20T15:36:52Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

+        Counter.builder()
+            .name("consumer_input_total")
+            .help("Total number of input messages")
+            .labelNames("type", "subtype")


I question most of these metrics really need type label. What is the cardinality of it and possible different combinations? I see in the test UPDATE is one. Is there also QUERY or something along those lines?

Yes, there's ADMIN and CONFIGSET.

Hmmm ok. I am not a fan of naming this label being called type. I think it should have some kind of context what it means as type and subtype can be very generic. Is it an operation or message_type maybe? Then what can subtype be? In core, I made it category but it is debateable if we should just remove that label/attribute all together from metrics. If you move type to something more specific then maybe you can just move off subtype to type. Again seeing an sample text output of these metrics would help if you can.

It's a request type - there are currently three types: UPDATE, ADMIN and CONFIGSET. Sub-type is primarily for UPDATE (add, dbi, dbq) and ADMIN (path).

Here's a sample output:

# HELP crossdc_consumer_input_total Total number of input messages # TYPE crossdc_consumer_input_total counter crossdc_consumer_input_total{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_total Total number of output requests # TYPE crossdc_consumer_output_total counter crossdc_consumer_output_total{otel_scope_name="org.apache.solr",result="handled",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_batch_size Histogram of output batch sizes # TYPE crossdc_consumer_output_batch_size histogram crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="5.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="10.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="25.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="50.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="75.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="100.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="250.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="750.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="1000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="2500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="5000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="7500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="10000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_batch_size_count{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1 crossdc_consumer_output_batch_size_sum{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_first_attempt_time_nanoseconds Histogram of first attempt request times # TYPE crossdc_consumer_output_first_attempt_time_nanoseconds histogram crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="25000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="50000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="100000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="250000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="500000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1000000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2500000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2.5E7"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1.0E8"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1.0E9"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_first_attempt_time_nanoseconds_count{otel_scope_name="org.apache.solr",type="UPDATE"} 1 crossdc_consumer_output_first_attempt_time_nanoseconds_sum{otel_scope_name="org.apache.solr",type="UPDATE"} 1.7667254470782164E18 # HELP crossdc_consumer_output_time_milliseconds Histogram of output request times # TYPE crossdc_consumer_output_time_milliseconds histogram crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="25.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="50.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="75.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="100.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="250.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="750.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="7500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_time_milliseconds_count{otel_scope_name="org.apache.solr",type="UPDATE"} 1 crossdc_consumer_output_time_milliseconds_sum{otel_scope_name="org.apache.solr",type="UPDATE"} 13.0 # TYPE target_info gauge target_info{service_name="unknown_service:java",telemetry_sdk_language="java",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.56.0"} 1

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

solr/test-framework/src/java/org/apache/solr/util/SolrKafkaTestsIgnoredThreadsFilter.java

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

…ecord counts.

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

gradle/libs.versions.toml

mlbiscoc · 2026-01-26T20:42:05Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+  public static final String ATTR_SUBTYPE = "subtype";
+  public static final String ATTR_RESULT = "result";
+
+  protected final Map<String, Attributes> attributesCache = new ConcurrentHashMap<>();


What is this for?

To avoid repeatedly creating millions of small objects (Attributes) when updating metrics.

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/Util.java

mlbiscoc

Just did another run through. Liking the changes. Just a few more comments.

mlbiscoc · 2026-01-29T19:55:50Z

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

          log.trace("result=nothandled_shutdown");
        }
-        metrics.counter(MetricRegistry.name(type.name(), "nothandled_shutdown")).inc();
+        metrics.incrementOutputCounter(type.name(), "nothandled_shutdown");


Maybe unhandled_shutdown instead of nothandled

mlbiscoc · 2026-01-29T20:03:58Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+  protected LongCounter inputMsg;
+  protected LongCounter inputReq;
+  protected LongCounter collapsed;
+  protected LongCounter output;
+  protected LongHistogram outputBatchSizeHistogram;
+  protected LongHistogram outputTimeHistogram;
+  protected LongHistogram outputBackoffHistogram;
+  protected LongHistogram outputFirstAttemptHistogram;


I had created this Attributed instrument wrappers so you could bind attributes to instruments which may have worked here and how all of Solr core does it. I probably should have mentioned that earlier but my fault. Honestly not a blocker and fine with this direction.

mlbiscoc · 2026-01-29T20:14:54Z

...-manager/src/java/org/apache/solr/crossdc/manager/messageprocessor/SolrMessageProcessor.java


    if (status != 0) {
-      metrics.counter(MetricRegistry.name(type.name(), "outputErrors")).inc();
+      metrics.incrementOutputCounter(type.name(), "solrError");


solr_error, or maybe just error. Should be known its in the context of Solr already

mlbiscoc · 2026-01-29T20:15:33Z

solr/cross-dc-manager/build.gradle

+//  implementation libs.prometheus.metrics.model
+//  implementation(libs.prometheus.metrics.expositionformats, {
+//    exclude group: "io.prometheus", module: "prometheus-metrics-shaded-protobuf"
+//    exclude group: "io.prometheus", module: "prometheus-metrics-config"
+//  })


Remove these comments

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

mlbiscoc · 2026-01-29T20:18:12Z

gradle/libs.versions.toml

+prometheus-metrics-core = { module = "io.prometheus:prometheus-metrics-core", version.ref = "prometheus-metrics" }
+prometheus-metrics-exporter-servlet-jakarta = { module = "io.prometheus:prometheus-metrics-exporter-servlet-jakarta", version.ref = "prometheus-metrics" }


Do we still need this? Otherwise just remove it.

…from #4063) (#4102)

sigram added 3 commits January 15, 2026 11:47

WIP.

a99dd06

SOLR-18060: Add Prometheus metrics to CrossDC Consumer.

aff9064

Add unit test, fix some bugs.

a42205a

sigram requested a review from mlbiscoc January 19, 2026 14:10

github-actions bot added dependencies Dependency upgrades test-framework tool:build tests labels Jan 19, 2026

sigram mentioned this pull request Jan 19, 2026

SOLR-18060: CrossDC Consumer - add Prometheus metrics. #4044

Closed

sigram added 4 commits January 20, 2026 13:25

Tidy.

5eef271

Add license and notice.

c569ff6

Fix a logging call.

82531d5

Remove unused licenses.

a7eee83

mlbiscoc reviewed Jan 20, 2026

View reviewed changes

Use OTEL API instead of Prometheus API.

13a8a53

github-actions bot removed the test-framework label Jan 22, 2026

sigram added 2 commits January 22, 2026 19:10

Fix first time latency calculation. Clean up metric names. Properly r…

b87ab4b

…ecord counts.

Tidy.

65fa041

github-actions bot added the cat:search label Jan 26, 2026

mlbiscoc reviewed Jan 26, 2026

View reviewed changes

sigram added 5 commits January 27, 2026 16:16

Fix issues from review. Add javadocs. Cleanups.

dc56222

Merge branch 'main' into jira/solr-18060-2

ee86a29

Resolve api drift.

ed8f582

Javadoc tidy.

97fdd65

Forbidden APIs.

c068dc1

sigram requested a review from mlbiscoc January 29, 2026 13:35

mlbiscoc reviewed Jan 29, 2026

View reviewed changes

Fix issues from review.

2f89366

sigram merged commit f1c4d25 into apache:main Feb 4, 2026
5 of 9 checks passed

sigram added a commit to sigram/solr that referenced this pull request Feb 4, 2026

SOLR-18060: Add Prometheus metrics to CrossDC Consumer. (apache#4063)

d947ca2

sigram added a commit that referenced this pull request Feb 4, 2026

SOLR-18060: Add Prometheus metrics to CrossDC Consumer. (cherry-pick …

4f9406a

…from #4063) (#4102)

		prometheus-metrics-core = { module = "io.prometheus:prometheus-metrics-core", version.ref = "prometheus-metrics" }
		prometheus-metrics-exporter-servlet-jakarta = { module = "io.prometheus:prometheus-metrics-exporter-servlet-jakarta", version.ref = "prometheus-metrics" }

Conversation

sigram commented Jan 19, 2026

Uh oh!

mlbiscoc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mlbiscoc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants